{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# MNIST Image Classification with TensorFlow\n",
    "\n",
    "This notebook demonstrates how to implement a simple linear image model on [MNIST](http://yann.lecun.com/exdb/mnist/) using the [tf.keras API](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras). It builds the foundation for this <a href=\"mnist_models.ipynb\">companion notebook</a>, which explores tackling the same problem with other types of models such as DNN and CNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook uses TF2.0\n",
    "Please check your tensorflow version using the cell below. If it is not 2.0, please run the pip line below and restart the kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.0.0-beta1\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensorflow==2.0.0b1\n"
     ]
    }
   ],
   "source": [
    "!pip freeze | grep tensorflow==2.0.0b1 || pip install tensorflow==2.0.0b1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "import unittest\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Dense, Flatten, Softmax\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint, TensorBoard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Exploring the data\n",
    "\n",
    "The MNIST dataset is already included in tensorflow through the keras datasets module. Let's load it and get a sense of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "mnist = tf.keras.datasets.mnist.load_data()\n",
    "(x_train, y_train), (x_test, y_test) = mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image height x width is 28 x 28\n",
      "There are 10 classes\n"
     ]
    }
   ],
   "source": [
    "HEIGHT, WIDTH = x_train[0].shape\n",
    "NCLASSES = tf.size(tf.unique(y_train).y)\n",
    "print(\"Image height x width is\", HEIGHT, \"x\", WIDTH)\n",
    "tf.print(\"There are\", NCLASSES, \"classes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each image is 28 x 28 pixels and represents a digit from 0 to 9. These images are black and white, so each pixel is a value from 0 (white) to 255 (black). Raw numbers can be hard to interpret sometimes, so we can plot the values to see the handwritten digit as an image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The label for image number 12 is 9\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADg5JREFUeJzt3XuMXPV5xvHnwawNmEtsaDYuuDFNKKlDywIb0whaSEgiYiUFqhZhqanT0jhSAyoVSYNADfxRKahtEkhLUU1wYyIuScrFboVSqGuJRCEuCzg2xlButrBlbBLT2EnA2N63f+xxtIGd36zndmZ5vx9pNTPnPWfOqyM/PjPzOzM/R4QA5HNI3Q0AqAfhB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+Q1KG93Nl0z4jDNLOXuwRSeU0/0+uxx5NZt63w2z5f0o2Spkn6WkRcX1r/MM3UmT6vnV0CKFgTqya9bssv+21Pk3STpI9Kmi9pke35rT4fgN5q5z3/AknPRsTzEfG6pLskXdCZtgB0WzvhP17Si+Meb6mW/RLbS2yP2B7Zqz1t7A5AJ3X90/6IWBoRwxExPKAZ3d4dgElqJ/xbJc0d9/iEahmAKaCd8D8i6STbJ9qeLukSSSs70xaAbmt5qC8i9tm+TNJ/amyob1lEbOhYZwC6qq1x/oi4X9L9HeoFQA9xeS+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBShB9IivADSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJtTVLr+1NknZL2i9pX0QMd6IpAN3XVvgrH4iIH3XgeQD0EC/7gaTaDX9IesD2o7aXdKIhAL3R7sv+syNiq+23S3rQ9lMR8dD4Far/FJZI0mE6os3dAeiUts78EbG1ut0h6V5JCyZYZ2lEDEfE8IBmtLM7AB3Ucvhtz7R91IH7kj4i6YlONQagu9p52T8o6V7bB57njoj4Tke6AtB1LYc/Ip6XdGoHewHQQwz1AUkRfiApwg8kRfiBpAg/kBThB5LqxLf60Mf2n3t6sX7oF7YX6/9+8spifcDTivW9sb9h7ay1lxS3PfaagWLdm7YW6z/++PyGtdn3la9HG929u1h/K+DMDyRF+IGkCD+QFOEHkiL8QFKEH0iK8ANJMc4/BXhG+ReQdv/+UMPatV9cVtz2nMN/XqyPFqvS3ijXRwvP8N2hO4rbnv43nyzWT31H+dy1Yt4/Nay9722XF7cd/MfvF+tvBZz5gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiApxvmngD3n/lax/t83NB7Pbmb1q0cW61/42z8r1gd+3mSgv2DXO8vnnunlSxD0158tX8Pwk9F9DWtHbmv8OwNZcOYHkiL8QFKEH0iK8ANJEX4gKcIPJEX4gaSajvPbXibpY5J2RMQp1bLZkr4paZ6kTZIujohXutfmW1u8vzzT+Rdv/peWn3vRcwuL9V3Xzi3WZ61+uOV9N3PMu08s1oe+/Vyx/pvTy+eu96z4q4a13/i3NcVtM5jMmf/rks5/w7KrJK2KiJMkraoeA5hCmoY/Ih6StPMNiy+QtLy6v1zShR3uC0CXtfqefzAitlX3X5I02KF+APRI2x/4RURIaniBt+0ltkdsj+zVnnZ3B6BDWg3/dttzJKm63dFoxYhYGhHDETE8oPIPUQLonVbDv1LS4ur+YkkrOtMOgF5pGn7bd0p6WNLJtrfYvlTS9ZI+bPsZSR+qHgOYQpqO80fEogal8zrcS1qvXPNqsX5Gk3dLC5/6g4a1aZ89urjttMcfKz95F/3fGeXPia99+7faev65D7S1+VseV/gBSRF+ICnCDyRF+IGkCD+QFOEHkuKnu3vghbt+u1jfcNq/Futb9pWHAg+5ZlbDWjy+rrhtt5WmF3/3FU8Wtz2kybnpTzeXR5sPv+9/ivXsOPMDSRF+ICnCDyRF+IGkCD+QFOEHkiL8QFKM8/fAn8wvjzeParRY37yv/LVc/aC+sfzSOL4kPX1D458lX/FrNxW3LR8VafPfn1ysHyF+nruEMz+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJMU4P4qmvbc8lr7x8mOK9ac+Xh7LL1n96pHF+lHff6FY39/ynnPgzA8kRfiBpAg/kBThB5Ii/EBShB9IivADSTUd57e9TNLHJO2IiFOqZddJ+pSkl6vVro6I+7vV5FR39wtDxfrnjl1frJ8242fF+u+ue+2ge5qsBUfcU6x/4PDyvpt9J7/kyh/+YbF+wvYNbTw7JnPm/7qk8ydY/pWIGKr+CD4wxTQNf0Q8JGlnD3oB0EPtvOe/zPY628tsN54vCkBfajX8N0t6l6QhSdskfanRiraX2B6xPbJXe1rcHYBOayn8EbE9IvZHxKikWyQtKKy7NCKGI2J4QOUfewTQOy2F3/accQ8vkvREZ9oB0CuTGeq7U9K5ko6zvUXStZLOtT0kKSRtkvTpLvYIoAscET3b2dGeHWe6PKf6W9EhRx1VrI/eV/5O/H+8Z0V5+7ZG09tzzucvL9ZHF/24Ye27Q3cUtz3/0r8o1qd/55FiPaM1sUq7Yqcnsy5X+AFJEX4gKcIPJEX4gaQIP5AU4QeS4qe7e2B09+7yCueV6x+8qDzkteOM1v8Pn7WxPNR7zO0/KNZf/kb5ku2nhu5qWLv1J/OK2x6xYVuxvq9YRTOc+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcb5p4Aj7l1TrM+7t0eNTOCpD36tWC993fimp88pbvurLz7ZUk+YHM78QFKEH0iK8ANJEX4gKcIPJEX4gaQIP5AU4/womvbek5us8Wixunnf6w1rg189rIWO0Cmc+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gqabj/LbnSrpN0qCkkLQ0Im60PVvSNyXNk7RJ0sUR8Ur3WkUdnr92elvb/9Hjf96w9o7Vj7X13GjPZM78+yRdGRHzJf2OpM/Yni/pKkmrIuIkSauqxwCmiKbhj4htEfFYdX+3pI2Sjpd0gaTl1WrLJV3YrSYBdN5Bvee3PU/SaZLWSBqMiAPzKb2ksbcFAKaISYff9pGS7pZ0RUTsGl+LiNDY5wETbbfE9ojtkb0qz+sGoHcmFX7bAxoL/u0RcU+1eLvtOVV9jqQdE20bEUsjYjgihgc0oxM9A+iApuG3bUm3StoYEV8eV1opaXF1f7GkFZ1vD0C3TOYrvWdJ+oSk9bbXVsuulnS9pG/ZvlTSZkkXd6dFdFO8/9RifeWZ/9zkGcpfy/WqWQfZEXqlafgj4nuS3KB8XmfbAdArXOEHJEX4gaQIP5AU4QeSIvxAUoQfSIqf7k5ux/tmFusnHloexy9NwS1Jh7424VXf6AOc+YGkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcb5k3vtuPI4fLNx/Bt2zi/Wj73l4YPuCb3BmR9IivADSRF+ICnCDyRF+IGkCD+QFOEHkmKcP7k/vnB1W9svW/GhYn2eGOfvV5z5gaQIP5AU4QeSIvxAUoQfSIrwA0kRfiCppuP8tudKuk3SoKSQtDQibrR9naRPSXq5WvXqiLi/W42iO+5+YahY/9yx63vUCXptMhf57JN0ZUQ8ZvsoSY/afrCqfSUi/qF77QHolqbhj4htkrZV93fb3ijp+G43BqC7Duo9v+15kk6TtKZadJntdbaX2Z7VYJsltkdsj+zVnraaBdA5kw6/7SMl3S3piojYJelmSe+SNKSxVwZfmmi7iFgaEcMRMTygGR1oGUAnTCr8tgc0FvzbI+IeSYqI7RGxPyJGJd0iaUH32gTQaU3Db9uSbpW0MSK+PG75nHGrXSTpic63B6BbJvNp/1mSPiFpve211bKrJS2yPaSx4b9Nkj7dlQ7RVbFqdrF+9QlnFuuDI/s72Q56aDKf9n9PkicoMaYPTGFc4QckRfiBpAg/kBThB5Ii/EBShB9IyhHlKZo76WjPjjN9Xs/2B2SzJlZpV+ycaGj+TTjzA0kRfiApwg8kRfiBpAg/kBThB5Ii/EBSPR3nt/2ypM3jFh0n6Uc9a+Dg9Gtv/dqXRG+t6mRv74yIX5nMij0N/5t2bo9ExHBtDRT0a2/92pdEb62qqzde9gNJEX4gqbrDv7Tm/Zf0a2/92pdEb62qpbda3/MDqE/dZ34ANakl/LbPt/207WdtX1VHD43Y3mR7ve21tkdq7mWZ7R22nxi3bLbtB20/U91OOE1aTb1dZ3trdezW2l5YU29zba+2/aTtDbb/slpe67Er9FXLcev5y37b0yT9r6QPS9oi6RFJiyLiyZ420oDtTZKGI6L2MWHbvyfpp5Jui4hTqmV/J2lnRFxf/cc5KyI+3ye9XSfpp3XP3FxNKDNn/MzSki6U9EnVeOwKfV2sGo5bHWf+BZKejYjnI+J1SXdJuqCGPvpeRDwkaecbFl8gaXl1f7nG/vH0XIPe+kJEbIuIx6r7uyUdmFm61mNX6KsWdYT/eEkvjnu8Rf015XdIesD2o7aX1N3MBAaradMl6SVJg3U2M4GmMzf30htmlu6bY9fKjNedxgd+b3Z2RJwu6aOSPlO9vO1LMfaerZ+GayY1c3OvTDCz9C/UeexanfG60+oI/1ZJc8c9PqFa1hciYmt1u0PSveq/2Ye3H5gktbrdUXM/v9BPMzdPNLO0+uDY9dOM13WE/xFJJ9k+0fZ0SZdIWllDH29ie2b1QYxsz5T0EfXf7MMrJS2u7i+WtKLGXn5Jv8zc3GhmadV87PpuxuuI6PmfpIUa+8T/OUnX1NFDg75+XdIPq78Ndfcm6U6NvQzcq7HPRi6VdKykVZKekfRfkmb3UW/fkLRe0jqNBW1OTb2drbGX9Oskra3+FtZ97Ap91XLcuMIPSIoP/ICkCD+QFOEHkiL8QFKEH0iK8ANJEX4gKcIPJPX/rJw9J1q+cE8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "IMGNO = 12\n",
    "# Uncomment to see raw numerical values.\n",
    "# print(x_test[IMGNO])\n",
    "plt.imshow(x_test[IMGNO].reshape(HEIGHT, WIDTH));\n",
    "print(\"The label for image number\", IMGNO, \"is\", y_test[IMGNO])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Define the model\n",
    "Let's start with a very simple linear classifier. This was the first method to be tried on MNIST in 1998, and scored an 88% accuracy. Quite ground breaking at the time!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can build our linear classifer using the [tf.keras API](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras), so we don't have to define or initialize our weights and biases. This happens automatically for us in the background. We can also add a softmax layer to transform the logits into probabilities. Finally, we can compile the model using categorical crossentropy in order to strongly penalize high probability predictions that were incorrect.\n",
    "\n",
    "When building more complex models such as DNNs and CNNs our code will be more readable by using the [tf.keras API](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras). Let's get one working so we can test it and use it as a benchmark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linear_model():\n",
    "    model = Sequential([\n",
    "        Flatten(),\n",
    "        Dense(NCLASSES),\n",
    "        Softmax()\n",
    "    ])\n",
    "\n",
    "    model.compile(optimizer='adam',\n",
    "                  loss='sparse_categorical_crossentropy',\n",
    "                  metrics=['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Write Input Functions\n",
    "\n",
    "As usual, we need to specify input functions for training and evaluating. We'll scale each pixel value so it's a decimal value between 0 and 1 as a way of normalizing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "BUFFER_SIZE = 5000\n",
    "BATCH_SIZE = 100\n",
    "\n",
    "TRAIN_KEY = tf.estimator.ModeKeys.TRAIN\n",
    "EVAL_KEY = tf.estimator.ModeKeys.EVAL\n",
    "\n",
    "\n",
    "def scale(image, label):\n",
    "    image = tf.cast(image, tf.float32)\n",
    "    image /= 255\n",
    "    return image, label\n",
    "\n",
    "\n",
    "def load_dataset(mode):\n",
    "    dataset = (x_train, y_train) if mode == TRAIN_KEY else (x_test, y_test)\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(dataset)\n",
    "    dataset = dataset.map(scale).batch(BATCH_SIZE)\n",
    "    if mode == TRAIN_KEY:\n",
    "        dataset = dataset.shuffle(BUFFER_SIZE).repeat()\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test for train passed!\n",
      "Test for eval passed!\n"
     ]
    }
   ],
   "source": [
    "def create_shape_test(key):\n",
    "    dataset = load_dataset(key)\n",
    "    data_iter = dataset.__iter__()\n",
    "    (images, labels) = data_iter.get_next()\n",
    "    expected_image_shape = (BATCH_SIZE, HEIGHT, WIDTH)\n",
    "    expected_label_shape = (BATCH_SIZE)\n",
    "    assert(images.shape == expected_image_shape)\n",
    "    assert(labels.shape == expected_label_shape)\n",
    "    print(\"Test for\", key, \"passed!\")\n",
    "\n",
    "\n",
    "create_shape_test(TRAIN_KEY)\n",
    "create_shape_test(EVAL_KEY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to train the model! The original MNIST linear classifier had an error rate of 12%. Let's use that to sanity check that our model is learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0918 21:42:53.459595 139790080771840 deprecation.py:323] From /home/jupyter/.local/lib/python3.5/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch 00001: saving model to mnist/linear_keras/checkpoints/mnist\n",
      "100/100 - 15s - loss: 1.2729 - accuracy: 0.5082 - val_loss: 0.7741 - val_accuracy: 0.8350\n",
      "Epoch 2/10\n",
      "\n",
      "Epoch 00002: saving model to mnist/linear_keras/checkpoints/mnist\n",
      "100/100 - 1s - loss: 0.6556 - accuracy: 0.8473 - val_loss: 0.5504 - val_accuracy: 0.8728\n",
      "Epoch 3/10\n",
      "\n",
      "Epoch 00003: saving model to mnist/linear_keras/checkpoints/mnist\n",
      "100/100 - 1s - loss: 0.5177 - accuracy: 0.8731 - val_loss: 0.4649 - val_accuracy: 0.8864\n",
      "Epoch 4/10\n",
      "\n",
      "Epoch 00004: saving model to mnist/linear_keras/checkpoints/mnist\n",
      "100/100 - 1s - loss: 0.4903 - accuracy: 0.8603 - val_loss: 0.4187 - val_accuracy: 0.8931\n",
      "Epoch 5/10\n",
      "\n",
      "Epoch 00005: saving model to mnist/linear_keras/checkpoints/mnist\n",
      "100/100 - 1s - loss: 0.4116 - accuracy: 0.8897 - val_loss: 0.3891 - val_accuracy: 0.9002\n",
      "Epoch 6/10\n",
      "\n",
      "Epoch 00006: saving model to mnist/linear_keras/checkpoints/mnist\n",
      "100/100 - 1s - loss: 0.4150 - accuracy: 0.8845 - val_loss: 0.3697 - val_accuracy: 0.9053\n",
      "Epoch 7/10\n",
      "\n",
      "Epoch 00007: saving model to mnist/linear_keras/checkpoints/mnist\n",
      "100/100 - 1s - loss: 0.3916 - accuracy: 0.8968 - val_loss: 0.3586 - val_accuracy: 0.9059\n",
      "Epoch 8/10\n",
      "\n",
      "Epoch 00008: saving model to mnist/linear_keras/checkpoints/mnist\n",
      "100/100 - 1s - loss: 0.3581 - accuracy: 0.9018 - val_loss: 0.3432 - val_accuracy: 0.9088\n",
      "Epoch 9/10\n",
      "\n",
      "Epoch 00009: saving model to mnist/linear_keras/checkpoints/mnist\n",
      "100/100 - 1s - loss: 0.3519 - accuracy: 0.9052 - val_loss: 0.3360 - val_accuracy: 0.9097\n",
      "Epoch 10/10\n",
      "\n",
      "Epoch 00010: saving model to mnist/linear_keras/checkpoints/mnist\n",
      "100/100 - 1s - loss: 0.3294 - accuracy: 0.9052 - val_loss: 0.3249 - val_accuracy: 0.9106\n"
     ]
    }
   ],
   "source": [
    "NUM_EPOCHS = 10\n",
    "STEPS_PER_EPOCH = 100\n",
    "\n",
    "model = linear_model()\n",
    "train_data = load_dataset(TRAIN_KEY)\n",
    "validation_data = load_dataset(EVAL_KEY)\n",
    "\n",
    "OUTDIR = \"mnist/linear_keras\"\n",
    "checkpoint_path = '{}/checkpoints/mnist'.format(OUTDIR)\n",
    "checkpoint_callback = ModelCheckpoint(\n",
    "    checkpoint_path, save_weights_only=True, verbose=1)\n",
    "tensorboard_callback = TensorBoard(log_dir=OUTDIR)\n",
    "\n",
    "history = model.fit(\n",
    "    train_data, \n",
    "    validation_data=validation_data,\n",
    "    epochs=NUM_EPOCHS, \n",
    "    steps_per_epoch=STEPS_PER_EPOCH,\n",
    "    verbose=2,\n",
    "    callbacks=[checkpoint_callback, tensorboard_callback]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test to beat benchmark accuracy passed!\n",
      "Test model accuracy is improving passed!\n",
      "Test loss is decreasing passed!\n"
     ]
    }
   ],
   "source": [
    "BENCHMARK_ERROR = .12\n",
    "BENCHMARK_ACCURACY = 1 - BENCHMARK_ERROR\n",
    "\n",
    "accuracy = history.history['accuracy']\n",
    "loss = history.history['loss']\n",
    "    \n",
    "assert(accuracy[-1] > BENCHMARK_ACCURACY)\n",
    "print(\"Test to beat benchmark accuracy passed!\")\n",
    "        \n",
    "assert(accuracy[0] < accuracy[1])\n",
    "assert(accuracy[1] < accuracy[-1])\n",
    "print(\"Test model accuracy is improving passed!\")\n",
    "    \n",
    "assert(loss[0] > loss[1])\n",
    "assert(loss[1] > loss[-1])\n",
    "print(\"Test loss is decreasing passed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Evaluating Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check Tensorboard to visually see how the model's accuracy and loss change over time. If running on a Deep Learning VM, wait for\n",
    "\n",
    "`TensorBoard <version> at http://<notebook name>:6006/`\n",
    "\n",
    "to appear, then go to File > New Launcher. Click on Tensorboard under \"Other\".\n",
    "Interrupt the kernel when you are ready to continue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"OUTDIR\"] = OUTDIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorBoard 1.14.0a20190603 at http://tf2-test:6006/ (Press CTRL+C to quit)\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!tensorboard --logdir=$OUTDIR --port=6006"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Were you able to get an accuracy of over 90%? Not bad for a linear estimator! Let's make some predictions and see if we can find where the model has trouble. Change the range of values below to find incorrect predictions, and plot the corresponding images. What would you have guessed for these images?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "image number: 8\n",
      "the prediction was 6\n",
      "the actual label is 5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "image_numbers = range(0, 10, 1)  # Change me, please.\n",
    "\n",
    "def load_prediction_dataset():\n",
    "    dataset = (x_test[image_numbers], y_test[image_numbers])\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(dataset)\n",
    "    dataset = dataset.map(scale).batch(len(image_numbers))\n",
    "    return dataset\n",
    "\n",
    "predicted_results = model.predict(load_prediction_dataset())\n",
    "for index, prediction in enumerate(predicted_results):\n",
    "    predicted_value = np.argmax(prediction)\n",
    "    actual_value = y_test[image_numbers[index]]\n",
    "    if actual_value != predicted_value:\n",
    "        print(\"image number: \" + str(image_numbers[index]))\n",
    "        print(\"the prediction was \" + str(predicted_value))\n",
    "        print(\"the actual label is \" + str(actual_value))\n",
    "        print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADmJJREFUeJzt3XGQVeV5x/Hfs8u6BILIhnQlgGIMjVKnYLJijE5rJ9VRQ4t2WqIzMTjjlLSNsXZsJpZ0pk77D9NpTE2bmJDAhDRKklGJZEJrDG0GM0bCYi0CK2gMBiiyUlTAJMvu8vSPPTgb3fPey73n3nOX5/uZ2dl7z3Pee565+uPce9+75zV3F4B42spuAEA5CD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAmNPNgZ1inT9TkZh4SCOVXel3HfcCq2beu8JvZNZLuldQu6avuviK1/0RN1qX2oXoOCSBhs2+set+aX/abWbukL0i6VtI8STeZ2bxaHw9Ac9Xznn+hpOfd/QV3Py7pm5IWF9MWgEarJ/wzJe0ddX9ftu3XmNkyM+s1s95BDdRxOABFavin/e6+0t173L2nQ52NPhyAKtUT/v2SZo+6PyvbBmAcqCf8WyTNNbPzzOwMSTdKWl9MWwAareapPncfMrPbJD2qkam+1e6+o7DOADRUXfP87r5B0oaCegHQRHy9FwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgmrpENzDahLO7k/Xjc9/VsGN37E6vL7Prb96drJ+1M70Kdlffr5L1tsf/O1lvBs78QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBUXfP8ZrZH0lFJw5KG3L2niKYwfrz20Q8k6/93Xf58910X/0dy7MfObNwC0KteOydZ/6Mp65L1aX8ysa7jL5r5/rrGF6GIL/n8nrsfKuBxADQRL/uBoOoNv0v6vpltNbNlRTQEoDnqfdl/hbvvN7PfkPSYmT3r7ptG75D9o7BMkiZqUp2HA1CUus787r4/+90vaZ2khWPss9Lde9y9p0Od9RwOQIFqDr+ZTTazKSdvS7pa0vaiGgPQWPW87O+WtM7MTj7OA+6enrsB0DJqDr+7vyBpfoG9oAHa5l+YrD/7ycnJ+uNX/3Oy/s72Lenjt+iE0q1Tf15hj/rm8ceD1vwvA6DhCD8QFOEHgiL8QFCEHwiK8ANBcenu09zr501J1ndfe1+FR3hbcc002Zdezb/89v0vXtLETt5qqp4v9fgSZ34gLMIPBEX4gaAIPxAU4QeCIvxAUIQfCIp5/iaYMGtmst736VnJevcT6eWgz1z7ZG6tbcCTY3cPHk/W9w6dlazPnvBqsn7L9qW5tVf63pEc270l3ftZT+xN1v3Ysdza1FfLn2cvG2d+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiKef4CtJ81NVlf+L2fJevfmb4+Wb+897ZT7umkzn9PX1r7Ux++JVkf3rErWW+/cG6y3rXrp/m1E7uTYysZqms0OPMDQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFAV5/nNbLWkRZL63f2ibFuXpG9JmiNpj6Ql7v5K49osX9vE/CWbBx5Mz/Mvn/6fyfp7H/6LZP2CdTuS9eFkNa3SPH7F8X3P1TUe5anmzP81Sde8adtdkja6+1xJG7P7AMaRiuF3902SDr9p82JJa7LbayRdX3BfABqs1vf83e5+ILv9kqTugvoB0CR1f+Dn7i4p92JrZrbMzHrNrHdQA/UeDkBBag3/QTObIUnZ7/68Hd19pbv3uHtPhzprPByAotUa/vWSTl6WdamkR4ppB0CzVAy/ma2V9GNJ7zWzfWZ2q6QVkq4ys+ck/X52H8A4UnGe391vyil9qOBeStU+bVqy/uw//GZubdeFX0yO3Vrho44L/v6FZH34yJH0AwA14Bt+QFCEHwiK8ANBEX4gKMIPBEX4gaC4dHfmfz96YbK+64Z/ya2tfz09Tbhq0VXJ+vDL+Ze3BhqFMz8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBMU8f+bopb+seey9P0v/dfPbdjOPj9bDmR8IivADQRF+ICjCDwRF+IGgCD8QFOEHgmKeP7P28pUV9sj/d/LBed9IjrzsnjuT9fPWH0/W23/4VLIO1IIzPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EVXGe38xWS1okqd/dL8q23S3pTyW9nO223N03NKrJZljY2ZGsD/pwbm1a28Tk2Gc/8oX0Yy/Jf2xJumjjnyXrU7fkH//YLE+OPTO9Orimb3s9vUMFh357cm6t+4f9ybHDXAehoao5839N0jVjbP+cuy/IfsZ18IGIKobf3TdJOtyEXgA0UT3v+W8zs21mttrM0utVAWg5tYb/PknnS1og6YCkz+btaGbLzKzXzHoHNVDj4QAUrabwu/tBdx929xOSviJpYWLfle7e4+49HeqstU8ABasp/GY2Y9TdGyRtL6YdAM1SzVTfWklXSppuZvsk/Z2kK81sgSSXtEfSxxvYI4AGMPf0PHCRzrQuv9TS17gvy+4vX5KuL/pSkzqJ4ycDlqzfsfPGZL1r0e4i2zktbPaNOuKH009shm/4AUERfiAowg8ERfiBoAg/EBThB4Jiqi9jE9JfeTh+5fzc2sf+9bvJsZPa0l9rXjTp5WS9w9qT9dPVCZ1I1n/rgduT9fM/9eMi2xkXmOoDUBHhB4Ii/EBQhB8IivADQRF+ICjCDwTFEt0ZHxpK1jt+sDW3tvaCd9V17M//cfpPV4c70tO2H/zrn+TWVpy9paaeWkFbhXPTrPkHmtTJ6YkzPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ExTx/C5j84Oa6xn93/mW5tRU3p+f5f+HHk/X3b/rzZP3cr6avNXDo9l/k1nov+UZyLBqLMz8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBFVxnt/MZkv6uqRuSS5ppbvfa2Zdkr4laY6kPZKWuPsrjWsVec55NLEuwM3psZPsjGS973dXJes3n3tVsr5hzqOJan3nnp+/1JWsz9Weuh7/dFfNsz8k6U53nyfpA5I+YWbzJN0laaO7z5W0MbsPYJyoGH53P+DuT2W3j0rqkzRT0mJJa7Ld1ki6vlFNAijeKb3uMrM5ki6WtFlSt7ufvI7SSxp5WwBgnKg6/Gb2dkkPSbrD3Y+MrvnIgn9jLvpnZsvMrNfMegeVXrMOQPNUFX4z69BI8O9394ezzQfNbEZWnyGpf6yx7r7S3XvcvadDnUX0DKAAFcNvZiZplaQ+d79nVGm9pKXZ7aWSHim+PQCNUnGJbjO7QtLjkp6R3lgzeblG3vd/W9I5kl7UyFTf4dRjtfIS3eNZ25QpubX+B2Ykxz75vrVFt1O1AR9M1hftTF/SfNKS9Mzy8KuvnXJP492pLNFdcZ7f3X8kKe/BSDIwTvENPyAowg8ERfiBoAg/EBThB4Ii/EBQXLr7NHDi6NHc2tmfnJYc+wer/zBZXz7ne8n6ZZ3DyfpDx6bn1j6z4SPJse/5qyeT9fSRUQlnfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IquLf8xeJv+cffw7e/sFk/eglv0zWL/jbQ7m1oRf31tQT8p3K3/Nz5geCIvxAUIQfCIrwA0ERfiAowg8ERfiBoPh7fiR1f/6JdL3C+KHiWkHBOPMDQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFAVw29ms83sv8xsp5ntMLO/zLbfbWb7zezp7Oe6xrcLoCjVfMlnSNKd7v6UmU2RtNXMHstqn3P3f2pcewAapWL43f2ApAPZ7aNm1idpZqMbA9BYp/Se38zmSLpY0uZs021mts3MVpvZmOtCmdkyM+s1s95BDdTVLIDiVB1+M3u7pIck3eHuRyTdJ+l8SQs08srgs2ONc/eV7t7j7j0d6iygZQBFqCr8ZtahkeDf7+4PS5K7H3T3YXc/IekrkhY2rk0ARavm036TtEpSn7vfM2r7jFG73SBpe/HtAWiUaj7tv1zSzZKeMbOns23LJd1kZgskuaQ9kj7ekA4BNEQ1n/b/SNJY1wHfUHw7AJqFb/gBQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCMndv3sHMXpb04qhN0yUdaloDp6ZVe2vVviR6q1WRvZ3r7u+sZsemhv8tBzfrdfee0hpIaNXeWrUvid5qVVZvvOwHgiL8QFBlh39lycdPadXeWrUvid5qVUpvpb7nB1Cess/8AEpSSvjN7Boz22Vmz5vZXWX0kMfM9pjZM9nKw70l97LazPrNbPuobV1m9piZPZf9HnOZtJJ6a4mVmxMrS5f63LXaitdNf9lvZu2Sdku6StI+SVsk3eTuO5vaSA4z2yOpx91LnxM2s9+RdEzS1939omzbP0o67O4rsn84p7n7p1ukt7slHSt75eZsQZkZo1eWlnS9pFtU4nOX6GuJSnjeyjjzL5T0vLu/4O7HJX1T0uIS+mh57r5J0uE3bV4saU12e41G/udpupzeWoK7H3D3p7LbRyWdXFm61Ocu0Vcpygj/TEl7R93fp9Za8tslfd/MtprZsrKbGUN3tmy6JL0kqbvMZsZQceXmZnrTytIt89zVsuJ10fjA762ucPf3SbpW0ieyl7ctyUfes7XSdE1VKzc3yxgrS7+hzOeu1hWvi1ZG+PdLmj3q/qxsW0tw9/3Z735J69R6qw8fPLlIava7v+R+3tBKKzePtbK0WuC5a6UVr8sI/xZJc83sPDM7Q9KNktaX0MdbmNnk7IMYmdlkSVer9VYfXi9paXZ7qaRHSuzl17TKys15K0ur5Oeu5Va8dvem/0i6TiOf+P9U0mfK6CGnr3dL+p/sZ0fZvUlaq5GXgYMa+WzkVknvkLRR0nOSfiCpq4V6+zdJz0jappGgzSiptys08pJ+m6Sns5/ryn7uEn2V8rzxDT8gKD7wA4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8Q1P8DiH5oG22VO38AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bad_image_number = 8\n",
    "plt.imshow(x_test[bad_image_number].reshape(HEIGHT, WIDTH));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's understandable why the poor computer would have some trouble. Some of these images are difficult for even humans to read. Ready for the next challenge? <a href=\"mnist_models.ipynb\">Click here</a> to super charge our models with human-like vision."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus Exercise\n",
    "\n",
    "Want to push your understanding further? Instead of using Keras'es built in layers, try repeating the above exercise with your own [custom layers](https://www.tensorflow.org/tutorials/customization/custom_layers)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<pre>\n",
    "# Copyright 2017 Google Inc. All Rights Reserved.\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#      http://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n",
    "</pre>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
